Systems and Methods for Debugging Neural Networks with Coverage Guided Fuzzing

ABSTRACT

The present disclosure provides systems and methods for debugging neural networks. In one example, a computer-implemented method is provided, which includes obtaining, by one or more computing devices, one or more inputs from an input corpus. The method further includes mutating, by the one or more computing devices, the one or more inputs and providing the one or more mutated inputs to a neural network; obtaining, by the one or more computing devices as a result of the neural network processing the one or more mutated inputs, a set of coverage arrays; determining, by the one or more computing devices based at least in part on the set of coverage arrays, whether the one or more mutated inputs provide new coverage; and upon determining that the one or more mutated inputs provide new coverage, adding the one or more mutated inputs to the input corpus.

The present application is based on and claims the benefit of U.S.Provisional Application No. 62/673,751 having a filing date of May 18,2018, which is incorporated by reference herein in its entirety for allpurposes.

FIELD

The present disclosure relates generally to machine learned models. Moreparticularly, the present disclosure relates to finding undesirablebehavior in neural networks.

BACKGROUND

The use of machine learning models, such as neural networks, is becomingmore important in solving a variety of tasks that have traditionallybeen difficult for a computing system. However, machine learning modelsare generally difficult to interpret and debug. As machine learningmodels like neural networks become more prevalent, it becomes moredesirable to test neural networks to discover bugs and/or otherundesirable behavior before a neural network is implemented in the “realworld.”

SUMMARY

Aspects and advantages of embodiments of the present disclosure will beset forth in part in the following description, or can be learned fromthe description, or can be learned through practice of the embodiments.

One example aspect of the present disclosure is directed to debugging aneural network. The method can include obtaining, by one or morecomputing devices, one or more inputs from an input corpus. The methodcan further include mutating, by the one or more computing devices, theone or more inputs. The method can further include providing, by the oneor more computing devices, the one or more mutated inputs to a neuralnetwork. The method can further include obtaining, by the one or morecomputing devices as a result of the neural network processing the oneor more mutated inputs, a set of coverage arrays that describe whetherone or more neurons of the neural network were activated duringprocessing of the one or more mutated inputs by the neural network. Themethod can further include determining, by the one or more computingdevices based at least in part on the set of coverage arrays, whetherthe one or more mutated inputs provide new coverage. The method canfurther include upon determining that the one or more mutated inputsprovide new coverage, adding, by the one or more computing devices, theone or more mutated inputs to the input corpus.

In some implementations, the method can further include obtaining, bythe one or more computing devices as a result of the neural networkprocessing the one or more mutated inputs, a set of metadata arrays thatdescribe metadata associated with execution of the neural network toprocess the one or more mutated inputs; determining, by the one or morecomputing devices based at least in part on the set of metadata arrays,whether an objective function is satisfied; and upon determining thatthe objective function is satisfied, adding, by the one or morecomputing devices, the one or more mutated inputs to a list of testcases.

Another example aspect of the present disclosure is directed to acomputing device. The computing device can include one or moreprocessors and one or more non-transitory computer-readable media thatstore instructions that, when executed by the one or more processors,cause the computing device to perform operations. The instructions, whenexecuted, can cause the computing device to obtain one or more inputsfrom an input corpus. The instructions, when executed, can further causethe computing device to mutate the one or more inputs. The instructions,when executed, can further cause the computing device to provide the oneor more mutated inputs to a neural network. The instructions, whenexecuted, can further cause the computing device to obtain as a resultof the neural network processing the one or more mutated inputs, a setof coverage arrays that describe whether one or more neurons of theneural network were activated during processing of the one or moremutated inputs by the neural network. The instructions, when executed,can further cause the computing device to determine based at least inpart on the set of coverage arrays, whether the one or more mutatedinputs provide new coverage. The instructions, when executed, canfurther cause the computing device to, upon determining that the one ormore mutated inputs provide new coverage, add the one or more mutatedinputs to the input corpus.

In some implementations, the computing device can further includeinstructions, that when executed, cause the computing device to obtainas a result of the neural network processing the one or more mutatedinputs, a set of metadata arrays that describe metadata associated withexecution of the neural network to process the one or more mutatedinputs; determine based at least in part on the set of metadata arrays,whether an objective function is satisfied; and upon determining thatthe objective function is satisfied, add the one or more mutated inputsto a list of test cases.

Another example aspect of the present disclosure is directed to one ormore non-transitory computer-readable media that store instructionsthat, when executed by one or more processors of a computing system,cause the computing system to perform operations. The operations includeobtaining one or more inputs from an input corpus. The operationsfurther include mutating the one or more inputs. The operations furtherinclude providing the one or more mutated inputs to a neural network.The operations further include obtaining, as a result of the neuralnetwork processing the one or more mutated inputs, a set of metadataarrays that describe metadata associated with execution of the neuralnetwork to process the one or more mutated inputs. The operationsfurther include determining, based at least in part on the set ofmetadata arrays, whether an objective function is satisfied. Theoperations further include upon determining that the objective functionis satisfied, add the one or more mutated inputs to a list of testcases.

In some implementations, the one or more non-transitorycomputer-readable media can store instructions that, when executed byone or more processors of a computing system, cause the computing systemto obtain, as a result of the neural network processing the one or moremutated inputs, a set of coverage arrays that describe whether one ormore neurons of the neural network were activated during processing ofthe one or more mutated inputs by the neural network; determine, basedat least in part on the set of coverage arrays, whether the one or moremutated inputs provide new coverage; and upon determining that the oneor more mutated inputs provide new coverage, add the one or more mutatedinputs to the input corpus.

BRIEF DESCRIPTION OF THE DRAWINGS

Detailed discussion of embodiments directed to one of ordinary skill inthe art is set forth in the specification, which makes reference to theappended figures, in which:

FIG. 1 depicts a block diagram of an example computing system that canbe used with machine learning models according to example embodiments ofthe present disclosure.

FIG. 2 depicts a block diagram of an example coverage guided fuzzingsystem according to example embodiments of the present disclosure.

FIG. 3 depicts a flow chart diagram of example operations to performneural network debugging according to example embodiments of the presentdisclosure.

Reference numerals that are repeated across plural figures are intendedto identify the same features in various implementations.

DETAILED DESCRIPTION

Overview

Generally, the present disclosure is directed to machine-learned models,such as neural networks. In particular, the systems and methods of thepresent disclosure can provide for testing neural networks to find bugsand/or other undesirable behavior, for example, before a neural networkis deployed. According to an aspect of the present disclosure, coverageguided fuzzing can be applied to neural networks to allow for debuggingof the neural networks. For example, in some implementations, coverageguided fuzzing can be applied to neural networks to provide for findingnumerical errors in trained neural networks, generating disagreementsbetween neural networks and quantized versions of those networks,surfacing undesirable behavior in models (e.g., character level languagemodels, etc.), and/or the like.

Machine learning models can be difficult to debug or interpret for avariety of reasons, ranging from the conceptual difficulty of specifyingwhat the user wishes to know about the model in formal terms tostatistical and computational difficulties in obtaining answers toformally specified questions. Neural networks can be particularlydifficult to debug because even relatively straightforward formalquestions about them can be computationally expensive to answer andbecause software implementations of neural networks can deviatesignificantly from theoretical models.

In general, coverage guided fuzzing provides for maintaining an inputcorpus comprising inputs to a program under consideration. Randomchanges are made to those inputs according to some mutation procedure,and the mutated inputs are added to an input corpus when they exercisenew coverage (e.g., causing code to execute in a different way thanpreviously seen, etc.).

The systems and methods of the present disclosure provide for inputtingrandom mutations of inputs to a neural network where the mutations areguided by a coverage metric toward the goal of satisfying user-specifiedconstraints. According to an aspect of the present disclosure, coveragecan be measured by analyzing the activation vectors of the neuralnetwork coverage graph. For example, in some implementations, newcoverage can be determined based on whether the neural network hasresulted in a state that the neural network has not reached previously,such that the new coverage helps to provide incremental progress indebugging.

As one example, fast approximate nearest neighbor algorithms can be usedto determine if two sets of neural network ‘activations’ aremeaningfully different from each other. This provides a coverage metricproducing useful results for neural networks, even when the underlyingimplementation of the neural network does not make use of manydata-dependent branches. For example, in some implementations, theactivations (or some subset of them) associated with each input can bestored and checked to determine whether coverage has increased on agiven input by using an approximate nearest neighbors algorithm to seewhether there are any other sets of activations within a pre-specifieddistance.

According to an aspect of the present disclosure, in someimplementations, coverage guided fuzzing of a neural network can startwith a seed corpus containing at least one set of inputs for thecomputation graph. The inputs can be restricted to those inputs that arein some sense valid neural network inputs. For example, if the inputsare images, the inputs can be restricted to those inputs that have thecorrect size and shape, and that lie in the same interval as the inputpixels of the dataset under consideration. As another example, if theinputs are sequences of characters, inputs can be restricted tocharacters that are in the vocabulary extracted from the training set.

Given this seed corpus, until instructed to stop or some other stoppingcriterion in met, the neural network debugging system can chooseelements from the input corpus according to some heuristic (e.g.,uniform random selection, some defined probability heuristic, etc.).Given this input, the neural network debugging system can perform somesort of modification to that input. For example, the modification can beas simple as just flipping the sign of an input pixel in an image.Additionally or alternatively, it can also be restricted to follow somekind of constraint on the total modification made to a corpus elementover time. The mutated inputs can then be fed to the neural network. Insome implementations, two things can be extracted from the neuralnetwork: a set of coverage arrays, from which the actual coverage can becomputed, and a set of metadata arrays, from which the result of theobjective function can be computed. For example, the coverage arrays candescribe which neurons of the neural network were activated duringprocessing of the input, and therefore may be referred to as or used togenerate an “activation vector.” As another example, the metadata arraycan describe a behavior, output, result, prediction, outcome, timings,statistics, run times, memory consumption, processor usage, and/or othermetadata associated with execution of the neural network to process theinput. Once the coverage and/or objective is computed, the mutated inputcan be added to the corpus if it exercises new coverage, and/or it canbe added to a list of test cases if it causes the objective function thebe satisfied.

For example, an objective function can be used to assess whether someparticular state (e.g., an erroneous state, etc.) has been reached. Theobjective function can be applied to the metadata arrays and inputs thatcaused the objective to be satisfied can be flagged. The neural networkdebugging system can determine whether the coverage provided by themutated input is new coverage (e.g., whether the neural network hasreached a state that it has not reached previously, etc.) based on thecoverage arrays. For example, in some implementations, when a newactivation vector is received, its nearest neighbor can be determined(e.g., through performance of an approximate nearest neighborsalgorithm) and checked for how far away the nearest neighbor is (e.g.,in Euclidean distance). The input can be added to the corpus if thedistance is greater than some defined amount.

In some implementations, the input mutations can be performed as a batchand the batch of inputs can be fed to the computation graph. Thecoverage and objective function can then be checked on a batch of outputarrays.

Reference now will be made in detail to embodiments, one or moreexamples of which are illustrated in the drawings. Each example isprovided by way of explanation of the embodiments, not limitation of thepresent disclosure. In fact, it will be apparent to those skilled in theart that various modifications and variations can be made to theembodiments without departing from the scope or spirit of the presentdisclosure. For instance, features illustrated or described as part ofone embodiment can be used with another embodiment to yield a stillfurther embodiment. Thus, it is intended that aspects of the presentdisclosure cover such modifications and variations.

Example Devices and Systems

FIG. 1 depicts a block diagram of an example computing system 100 thatprovides for the use of machine learning according to exampleembodiments of the present disclosure. The system 100 includes a usercomputing device 102, a server computing system 130, and a trainingcomputing system 150 that are communicatively coupled over a network180.

The user computing device 102 can be any type of computing device, suchas, for example, a personal computing device (e.g., laptop or desktop),a mobile computing device (e.g., smartphone or tablet), a gaming consoleor controller, a wearable computing device, an embedded computingdevice, or any other type of computing device.

The user computing device 102 includes one or more processors 112 and amemory 114. The one or more processors 112 can be any suitableprocessing device (e.g., a processor core, a microprocessor, an ASIC, aFPGA, a controller, a microcontroller, etc.) and can be one processor ora plurality of processors that are operatively connected. The memory 114can include one or more non-transitory computer-readable storagemediums, such as RAM, ROM, EEPROM, EPROM, flash memory devices, magneticdisks, etc., and combinations thereof. The memory 114 can store data 116and instructions 118 which are executed by the processor 112 to causethe user computing device 102 to perform operations.

In some implementations, the user computing device 102 can store orinclude one or more machine-learned models 120. For example, themachine-learned models 120 can be or can otherwise include variousmachine-learned models such as neural networks (e.g., deep neuralnetworks) or other types of machine-learned models, including non-linearmodels and/or linear models. Neural networks can include feed-forwardneural networks, recurrent neural networks (e.g., long short-term memoryrecurrent neural networks), convolutional neural networks or other formsof neural networks.

In some implementations, the one or more machine-learned models 120 canbe received from the server computing system 130 over network 180,stored in the user computing device memory 114, and then used orotherwise implemented by the one or more processors 112. In someimplementations, the user computing device 102 can implement multipleparallel instances of a single machine-learned model 120.

Additionally or alternatively, one or more machine-learned models 140can be included in or otherwise stored and implemented by the servercomputing system 130 that communicates with the user computing device102 according to a client-server relationship. For example, themachine-learned models 140 can be implemented by the server computingsystem 140 as a portion of a cloud based service. Thus, one or moremodels 120 can be stored and implemented at the user computing device102 and/or one or more models 140 can be stored and implemented at theserver computing system 130.

The user computing device 102 can also include one or more user inputcomponent 122 that receives user input. For example, the user inputcomponent 122 can be a touch-sensitive component (e.g., atouch-sensitive display screen or a touch pad) that is sensitive to thetouch of a user input object (e.g., a finger or a stylus). Thetouch-sensitive component can serve to implement a virtual keyboard.Other example user input components include a microphone, a traditionalkeyboard, or other means by which a user can provide user input.

The server computing system 130 includes one or more processors 132 anda memory 134. The one or more processors 132 can be any suitableprocessing device (e.g., a processor core, a microprocessor, an ASIC, aFPGA, a controller, a microcontroller, etc.) and can be one processor ora plurality of processors that are operatively connected. The memory 134can include one or more non-transitory computer-readable storagemediums, such as RAM, ROM, EEPROM, EPROM, flash memory devices, magneticdisks, etc., and combinations thereof. The memory 134 can store data 136and instructions 138 which are executed by the processor 132 to causethe server computing system 130 to perform operations.

In some implementations, the server computing system 130 includes or isotherwise implemented by one or more server computing devices. Ininstances in which the server computing system 130 includes pluralserver computing devices, such server computing devices can operateaccording to sequential computing architectures, parallel computingarchitectures, or some combination thereof.

As described above, the server computing system 130 can store orotherwise include one or more machine-learned models 140. For example,the models 140 can be or can otherwise include various machine-learnedmodels. Example machine-learned models include neural networks or othermulti-layer non-linear models. Example neural networks include feedforward neural networks, deep neural networks, recurrent neuralnetworks, and convolutional neural networks.

In some implementations, the server computing system 130 can furtherinclude a neural network debugging system 142, such as described hereinwith regard to FIG. 2. For example, the neural network debugging system142 can provide for performing coverage guided fuzzing using a corpus ofinputs, for instance, to provide for testing neural networks, such as todiscover errors which may occur for rare inputs.

The user computing device 102 and/or the server computing system 130 cantrain the models 120 and/or 140 via interaction with the trainingcomputing system 150 that is communicatively coupled over the network180. The training computing system 150 can be separate from the servercomputing system 130 or can be a portion of the server computing system130.

The training computing system 150 includes one or more processors 152and a memory 154. The one or more processors 152 can be any suitableprocessing device (e.g., a processor core, a microprocessor, an ASIC, aFPGA, a controller, a microcontroller, etc.) and can be one processor ora plurality of processors that are operatively connected. The memory 154can include one or more non-transitory computer-readable storagemediums, such as RAM, ROM, EEPROM, EPROM, flash memory devices, magneticdisks, etc., and combinations thereof. The memory 154 can store data 156and instructions 158 which are executed by the processor 152 to causethe training computing system 150 to perform operations. In someimplementations, the training computing system 150 includes or isotherwise implemented by one or more server computing devices.

The training computing system 150 can include a model trainer 160 thattrains the machine-learned models 120 and/or 140 stored at the usercomputing device 102 and/or the server computing system 130 usingvarious training or learning techniques, such as, for example, backwardspropagation of errors. In some implementations, performing backwardspropagation of errors can include performing truncated backpropagationthrough time. The model trainer 160 can perform a number ofgeneralization techniques (e.g., weight decays, dropouts, etc.) toimprove the generalization capability of the models being trained. Inparticular, the model trainer 160 can train the machine-learned models120 and/or 140 based on a set of training data 162.

In some implementations, if the user has provided consent, the trainingexamples can be provided by the user computing device 102. Thus, in suchimplementations, the model 120 provided to the user computing device 102can be trained by the training computing system 150 on user-specificdata received from the user computing device 102. In some instances,this process can be referred to as personalizing the model.

The model trainer 160 includes computer logic utilized to providedesired functionality. The model trainer 160 can be implemented inhardware, firmware, and/or software controlling a general purposeprocessor. For example, in some implementations, the model trainer 160includes program files stored on a storage device, loaded into a memoryand executed by one or more processors. In other implementations, themodel trainer 160 includes one or more sets of computer-executableinstructions that are stored in a tangible computer-readable storagemedium such as RAM hard disk or optical or magnetic media.

The network 180 can be any type of communications network, such as alocal area network (e.g., intranet), wide area network (e.g., Internet),or some combination thereof and can include any number of wired orwireless links. In general, communication over the network 180 can becarried via any type of wired and/or wireless connection, using a widevariety of communication protocols (e.g., TCP/IP, HTTP, SMTP, FTP),encodings or formats (e.g., HTML, XML), and/or protection schemes (e.g.,VPN, secure HTTP, SSL).

FIG. 1 illustrates one example computing system that can be used toimplement the present disclosure. Other computing systems can be used aswell. For example, in some implementations, the user computing device102 can include the model trainer 160 and the training dataset 162. Insuch implementations, the models 120 can be both trained and usedlocally at the user computing device 102. In some of suchimplementations, the user computing device 102 can implement the modeltrainer 160 to personalize the models 120 based on user-specific data.

Example Debugging System Arrangement

FIG. 2 depicts a block diagram of an example neural network debuggingsystem 200 using coverage guided fuzzing according to exampleembodiments of the present disclosure. In some implementations, theneural network debugging system 200 can provide for performing coverageguided fuzzing using a corpus of inputs, for example, to provide fortesting neural networks, such as to discover errors which may occur forrare inputs. The neural network debugging system 200 can allow forguiding mutations to corpus inputs by a coverage metric to work toward agoal of satisfying user-specified constraints (e.g., random changes aremade to inputs according to some mutation procedure and the mutatedinputs are added to an input corpus when they exercise new coverage). Asan example, coverage can be measured by analyzing the activation vectorsof the neural network coverage graph. For instance, in someimplementations, new coverage can be determined based on whether theneural network has resulted in a state that the neural network has notreached previously, such that the new coverage helps to provideincremental progress in debugging of the neural network model. Forexample, in some implementations, coverage guided fuzzing can be appliedto neural networks to provide for finding numerical errors in trainedneural networks, generating disagreements between neural networks andquantized versions of those networks, surfacing undesirable behavior inmodels, and/or the like.

As illustrated in FIG. 2, a neural network debugging system 200 caninclude a coverage guided fuzzer 202 and a seed corpus 220 (e.g.,containing at least one set of inputs for the computation graph) whichcan provide for an initial set of inputs to a coverage guided fuzzer 202to test a neural network.

The coverage guided fuzzer 202 can obtain (e.g., select) a set of inputsfrom the seed corpus 220 to provide an input corpus 204, which maycomprise all or some subset of inputs included in the seed corpus 220.In some implementations, the inputs can be restricted to some type ofvalid neural network inputs (e.g., images having a correct size andshape, characters that are in a vocabulary extracted from a trainingset, etc.). In some implementations, the seed corpus 220 can be suppliedby a user and/or can be selected from a set of available seed corpuses.The inputs can be textual inputs, image inputs, audio data inputs,sensor data inputs, and/or various other types of inputs.

The coverage guided fuzzer 202 can include an input chooser 206 that canselect input(s) from the input corpus 204 to use during a particulariteration of the coverage guided fuzzing. For example, in someimplementations, the input chooser 206 can select inputs using uniformrandom selection. For example, in some implementations, the inputchooser 206 can be biased towards selecting inputs that were morerecently added to the input corpus 204. As one example, the inputchooser 206 can select inputs using a heuristic such as

${{p\left( {c_{k},t} \right)} = \frac{e^{t_{k} - t}}{\sum e^{t_{k} - t}}},$

wherein p(c_(k), t) gives a probability of choosing an input corpuselement c_(k) at time t where t_(k) is the time when element c_(k) wasadded to the input corpus. The intuition behind this is that recentlysampled inputs are more likely to yield useful new coverage whenmutated, but that this advantage decays as time progresses, and thusinputs can be selected as a function of their age.

The input chooser 206 can provide the selected input(s) to a mutator208. The mutator 208 can apply modifications (e.g., mutations) to theselected input(s) before the inputs are provided to the neural network.For example, in some implementations, the mutator 208 can add whitenoise of a user-configurable variance to input(s) (e.g., image inputs,etc.). As another example, in some implementations, the mutator 208 canadd white noise of a user-configurable variance to the one or moreinputs (e.g., image inputs, etc.), wherein a difference between themutated input and an original input from which the mutated input isdescended is constrained to have a user-configurable L_(∞) norm. Thistype of constrained mutation can be useful to find inputs that satisfysome objective function, but are still plausibly of the same “class” asthe original input that was used as a seed. In some implementations, theimage can be clipped after mutation so that it lies in the same range asthe inputs used to train the neural network being debugged.

As another example, in some implementations, such as with text stringinputs, one of a set of operations can be uniformly performed at random,including operations such as deleting a character at a random location,adding a character at a random location, substituting a random characterat a random location, and/or the like.

The input chooser 206 includes computer logic utilized to providedesired functionality. The input chooser 206 can be implemented inhardware, firmware, and/or software controlling a general purposeprocessor. For example, in some implementations, the input chooser 206includes program files stored on a storage device, loaded into a memoryand executed by one or more processors. In other implementations, theinput chooser 206 includes one or more sets of computer-executableinstructions that are stored in a tangible computer-readable storagemedium such as RAM hard disk or optical or magnetic media.

The mutator 208 can then provide the mutated input(s) to the neuralnetwork 210. The neural network 210 can provide outputs which caninclude a set of coverage arrays, for which coverage can be computed,and a set of metadata arrays, from which a result of an objectivefunction can be computed. For example, when the mutated inputs are fedinto a computation graph, both coverage arrays and metadata arrays arereturned as output.

The mutator 208 includes computer logic utilized to provide desiredfunctionality. The mutator 208 can be implemented in hardware, firmware,and/or software controlling a general purpose processor. For example, insome implementations, the mutator 208 includes program files stored on astorage device, loaded into a memory and executed by one or moreprocessors. In other implementations, the mutator 208 includes one ormore sets of computer-executable instructions that are stored in atangible computer-readable storage medium such as RAM hard disk oroptical or magnetic media.

The objective function 212 can assess whether the neural network hasreached some particular state, for example, a state which may beregarded as erroneous, based on the metadata array(s). An erroneousstate may include an incorrect prediction, an execution time greaterthan a maximum execution time, a processor usage greater than a maximumprocessor usage, a failure of the neural network to execute, and/orother the existence of other errors or undesirable behavior orperformance. In some implementations, the objective function 212 can bespecified by a user and/or selected from a set of available objectivefunctions. Generally, the objective function 212 used to assess whetherthe neural network has reached some particular state can be separate anddistinct from some other objective or loss function used to train theneural network. If the objective function 212 is satisfied, the mutatedinput(s) provided to the neural network can be flagged, such as beingadded to a list of test cases (e.g., for future debugging, etc.). As anexample, when the mutated inputs are fed into a computation graph andmetadata arrays are returned as output, the objective function can beapplied to the metadata arrays and any mutated inputs that caused theobjective function to be satisfied can be flagged.

The coverage analyzer 214 can determine whether the coverage provided bythe mutated input(s) is new coverage (e.g., whether the neural networkhas reached a state that it has not reached previously, etc.) based onthe coverage array(s). For example, in some implementations, coverageanalyzer 214 can determine whether new coverage is provided based onwhether an activation vector is approximately close to a previousactivation vector. If the coverage analyzer 214 determines that themutated input(s) provide new coverage, the mutated input(s) can be addedto the input corpus 204, for example, to be used as input(s) in futureiterations of debugging and/or the like. For example, an approximatenearest neighbor can be computed for a new activation vector and checkedto determine how far away the nearest neighbor is in Euclidean distancefrom the activation vector. The input can be added to the corpus if thedistance is greater than some defined amount (e.g., which can be auser-configurable hyperparameter, an adaptive hyperparameter to adaptsover time, and/or a dynamic hyperparameter that changes over time, forexample, according to a predetermined schedule). In someimplementations, the coverage guided fuzzer 202 can continue to select,mutate, and analyze inputs included in the input corpus 204 untilinstructed to stop and/or some other stopping criterion in met.

The coverage analyzer 214 includes computer logic utilized to providedesired functionality. The coverage analyzer 214 can be implemented inhardware, firmware, and/or software controlling a general purposeprocessor. For example, in some implementations, the coverage analyzer214 includes program files stored on a storage device, loaded into amemory and executed by one or more processors. In other implementations,the coverage analyzer 214 includes one or more sets ofcomputer-executable instructions that are stored in a tangiblecomputer-readable storage medium such as RAM hard disk or optical ormagnetic media.

The coverage arrays and/or associated activation vectors may describewhether some or all of the neurons of the neural network were activatedduring processing of an input. As one example, a coverage array and/orassociated activation vector may be limited to describing only whetherthe logits of the neural network and/or neurons of a layer of thenetwork prior to the logits were activated.

In some implementations, the system 200 can be applied (e.g., inparallel) to two or more different (but potentially related) models toidentify disagreements between the models. For example, the two or moredifferent models can be two or more different versions of a base modelsuch as a base model and a quantized version of the base model. Toidentify disagreements, the same input (e.g., a mutated input) can beprovided to the two or more different models and the two or moredifferent outputs of the two or more different models can be analyzed(e.g., according to the objective function 212 and/or the coverageanalyzer 214) to detect disagreements or otherwise measure a divergencein the outputs.

Example Methods

FIG. 3 depicts a flow chart diagram of example operations to performneural network debugging according to example embodiments of the presentdisclosure. Although FIG. 3 depicts steps performed in a particularorder for purposes of illustration and discussion, the methods of thepresent disclosure are not limited to the particularly illustrated orderor arrangement. The various steps of the method 300 can be omitted,rearranged, combined, and/or adapted in various ways without deviatingfrom the scope of the present disclosure.

At 302, a computing system can obtain an input corpus, for example froma seed corpus comprising one or more sets of inputs. For example, a seedcorpus can contain at least one set of inputs for the computation graph.The inputs can be restricted to those inputs that are in some sensevalid neural network inputs. For example, if the inputs are images, theinputs can be restricted to those inputs that have the correct size andshape, and that lie in the same interval as the input pixels of thedataset under consideration. As another example, if the inputs aresequences of characters, inputs can be restricted to characters that arein the vocabulary extracted from the training set.

At 304, the computing system can select one or more inputs from theinput corpus for use in debugging a neural network. For example, thecomputing system can select one or more inputs from the input corpusbased on uniform random selection, based on one or more heuristics(e.g.,

${p\left( {c_{k},t} \right)} = \frac{e^{t_{k} - t}}{\sum e^{t_{k} - t}}$

giving a probability of choosing an input corpus element c_(k) at time twhere t_(k) is the time when element c_(k) was added to the inputcorpus, etc.), and/or the like.

At 306, the computing system can modify the selected input(s) prior toinput to the neural network by performing some type of mutation on theselected input(s). For example, in some implementations, the computingsystem can perform a simple modification of the input such as flipping asign of an input. As another example, in some implementations, computingsystem can restrict the modifications to follow a constraint on thetotal modification made to a corpus element over time.

At 308, the computing system feed the modified input(s) to the neuralnetwork that is to be debugged.

At 310, the computing system can obtain, as a result of the neuralnetwork processing the one or more mutated inputs, a set of coveragearrays (e.g., that describe whether one or more neurons of the neuralnetwork were activated during processing of the one or more mutatedinputs by the neural network) which can be used to compute the actualcoverage exercised by the modified input(s).

At 312, the computing system the computing system can determine whetherthe mutated input(s) provide new coverage at least in part on thecoverage array(s). For example, the computing system can determine thatnew coverage is provided is the neural network results in a state thatit has not been in before. If the mutated input(s) provide new coverage,operation continues to 314. If the mutated input(s) do not provide newcoverage, operations continue to 322, where a next input can beanalyzed. For example, in some implementations, when a new activationvector is received, its nearest neighbor can be determined and checkedfor how far away the nearest neighbor is in Euclidean distance. Theinput can be added to the corpus if the distance is greater than somedefined amount.

At 314, the computing system can add the mutated input(s) to the inputcorpus.

At 316, the computing system, as a result of the neural networkprocessing the one or more mutated inputs, a set of metadata arrays(e.g., that describe metadata associated with execution of the neuralnetwork to process the one or more mutated inputs) for use in computingthe objective function.

At 318, the computing system can determine whether the objectivefunction is satisfied based at least in part on the metadata array(s).For example, the objective function can assess whether the neuralnetwork has reached a particular state, such as state that is regardedas erroneous. For example, the objective function can be applied to themetadata arrays and inputs that cause the objective to be satisfied canbe flagged. If the objective function is satisfied, operation continuesto 320. If the objective function is not satisfied, operation continuesto 322.

At 320, the computing system can add the mutated input to a list of testcases.

Additional Disclosure

The technology discussed herein makes reference to servers, databases,software applications, and other computer-based systems, as well asactions taken and information sent to and from such systems. Theinherent flexibility of computer-based systems allows for a greatvariety of possible configurations, combinations, and divisions of tasksand functionality between and among components. For instance, processesdiscussed herein can be implemented using a single device or componentor multiple devices or components working in combination. Databases andapplications can be implemented on a single system or distributed acrossmultiple systems. Distributed components can operate sequentially or inparallel.

While the present subject matter has been described in detail withrespect to various specific example embodiments thereof, each example isprovided by way of explanation, not limitation of the disclosure. Thoseskilled in the art, upon attaining an understanding of the foregoing,can readily produce alterations to, variations of, and equivalents tosuch embodiments. Accordingly, the subject disclosure does not precludeinclusion of such modifications, variations and/or additions to thepresent subject matter as would be readily apparent to one of ordinaryskill in the art. For instance, features illustrated or described aspart of one embodiment can be used with another embodiment to yield astill further embodiment. Thus, it is intended that the presentdisclosure cover such alterations, variations, and equivalents.

1.-20. (canceled)
 21. A computer-implemented method for debugging aneural network, the method comprising: obtaining, by one or morecomputing devices, one or more inputs from an input corpus; mutating, bythe one or more computing devices, the one or more inputs to generateone or more mutated inputs, wherein the one or more mutated inputs areconstrained to have a same class as the one or more inputs; providing,by the one or more computing devices, the one or more mutated inputs toa neural network; obtaining, by the one or more computing devices as aresult of the neural network processing the one or more mutated inputs,a set of coverage arrays that describe whether one or more neurons ofthe neural network were activated during processing of the one or moremutated inputs by the neural network; determining, by the one or morecomputing devices based at least in part on the set of coverage arrays,whether the one or more mutated inputs provide new coverage; and upondetermining that the one or more mutated inputs provide new coverage,adding, by the one or more computing devices, the one or more mutatedinputs to the input corpus.
 22. The method of claim 21, furthercomprising: obtaining, by the one or more computing devices as a resultof the neural network processing the one or more mutated inputs, a setof metadata arrays that describe metadata associated with execution ofthe neural network to process the one or more mutated inputs;determining, by the one or more computing devices based at least in parton the set of metadata arrays, whether an objective function issatisfied; and upon determining that the objective function issatisfied, adding, by the one or more computing devices, the one or moremutated inputs to a list of test cases.
 23. The method of claim 21,wherein determining, by the one or more computing devices based at leastin part on the set of coverage arrays, whether the one or more mutatedinputs provide new coverage comprises: generating, by the one or morecomputing devices, an activation vector based at least in part on theset of coverage arrays; performing, by the one or more computingdevices, an approximate nearest neighbors algorithm to identify aprevious activation vector; determining, by the one or more computingdevices, a distance between the activation vector and the previousactivation vector identified by the approximate nearest neighborsalgorithm; and comparing, by the one or more computing devices, thedistance to a threshold distance; wherein the one or more mutated inputsprovide new coverage when the distance is greater than the thresholddistance.
 24. The method of claim 21, wherein obtaining one or moreinputs from the input corpus comprises using uniform random selection toselect the one or more inputs from the input corpus.
 25. The method ofclaim 21, wherein obtaining one or more inputs from the input corpuscomprises selecting the one or more inputs from the input corpus using aheuristic of${{p\left( {c_{k},t} \right)} = \frac{e^{t_{k} - t}}{\sum e^{t_{k} - t}}},$wherein p(c_(k), t) gives a probability of choosing input corpus elementc_(k) at time t where t_(k) is the time when element c_(k) was added tothe input corpus.
 26. The method of claim 21, wherein mutating the oneor more inputs comprises mutating the one or more inputs subject to aconstraint, wherein the constraint comprises a L_(∞) norm.
 27. Themethod of claim 26, wherein the L_(∞) norm is specified by a user. 28.The method of claim 21, wherein determining whether the one or moremutated inputs provide new coverage comprises determining whether theneural network has reached a new state that it has not previouslyreached.
 29. The method of claim 28, wherein determining whether theneural network has reached a new state that it has not previouslyreached comprises determining whether an activation vector isapproximately close to a previous activation vector.
 30. The method ofclaim 22, wherein determining whether the objective function issatisfied comprises determining whether the neural network has reached adesired state.
 31. The method of claim 30, wherein the desired state isan erroneous state for the neural network.
 32. A computing devicecomprising: one or more processors; and one or more non-transitorycomputer-readable media that store instructions that, when executed bythe one or more processors, cause the computing device to: obtain one ormore inputs from an input corpus; mutate the one or more inputs togenerate one or more mutated inputs, wherein the one or more mutatedinputs are constrained to have a same class as the one or more inputs;provide the one or more mutated inputs to a neural network; obtain as aresult of the neural network processing the one or more mutated inputs,a set of coverage arrays that describe whether one or more neurons ofthe neural network were activated during processing of the one or moremutated inputs by the neural network; determine based at least in parton the set of coverage arrays, whether the one or more mutated inputsprovide new coverage; and upon determining that the one or more mutatedinputs provide new coverage, add the one or more mutated inputs to theinput corpus.
 33. The computing device of claim 32, further comprisinginstructions, that when executed, cause the computing device to: obtainas a result of the neural network processing the one or more mutatedinputs, a set of metadata arrays that describe metadata associated withexecution of the neural network to process the one or more mutatedinputs; determine based at least in part on the set of metadata arrays,whether an objective function is satisfied; and upon determining thatthe objective function is satisfied, add the one or more mutated inputsto a list of test cases.
 34. The computing device of claim 32, furthercomprising instructions, that when executed, cause the computing deviceto: obtain the input corpus from a seed corpus, the seed corpuscontaining at least one set of inputs.
 35. The computing device of claim32, wherein obtaining one or more inputs from the input corpuscomprises: using uniform random selection to select the one or moreinputs from the input corpus; or selecting the one or more inputs fromthe input corpus using a heuristic of${{p\left( {c_{k},t} \right)} = \frac{e^{t_{k} - t}}{\sum e^{t_{k} - t}}},$wherein p(c_(k), t) gives a probability of choosing input corpus elementc_(k) at time t where t_(k) is the time when element c_(k) was added tothe input corpus.
 36. The computing device of claim 32, wherein adifference between the one or more mutated inputs and the one or moreinputs from which the one or more mutated inputs are descended isconstrained to have a L_(∞) norm.
 37. The computing device of claim 32,wherein determining whether the one or more mutated inputs provide newcoverage comprises determining whether the neural network has reached anew state that it has not previously reached; and wherein determiningwhether the neural network has reached a new state that it has notpreviously reached comprises determining whether an activation vector isapproximately close to a previous activation vector.
 38. The computingdevice of claim 33, wherein determining whether the objective functionis satisfied comprises determining whether the neural network hasreached a desired state, wherein the desired state is an erroneous statefor the neural network.
 39. One or more non-transitory computer-readablemedia that store instructions that, when executed by one or moreprocessors of a computing system, cause the computing system to performoperations, the operations comprising: obtaining one or more inputs froman input corpus; mutating the one or more inputs to generate one or moremutated inputs, wherein a difference between the one or more mutatedinputs and the one or more inputs from which the one or more mutatedinputs are descended is constrained to have a L_(∞) norm; providing theone or more mutated inputs to a neural network; obtaining, as a resultof the neural network processing the one or more mutated inputs, a setof metadata arrays that describe metadata associated with execution ofthe neural network to process the one or more mutated inputs;determining, based at least in part on the set of metadata arrays,whether an objective function is satisfied; and upon determining thatthe objective function is satisfied, adding the one or more mutatedinputs to a list of test cases.
 40. The one or more non-transitorycomputer-readable media of claim 39, wherein the operations furthercomprise: obtaining, as a result of the neural network processing theone or more mutated inputs, a set of coverage arrays that describewhether one or more neurons of the neural network were activated duringprocessing of the one or more mutated inputs by the neural network;determining, based at least in part on the set of coverage arrays,whether the one or more mutated inputs provide new coverage; and upondetermining that the one or more mutated inputs provide new coverage,adding the one or more mutated inputs to the input corpus.